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This report is inteifded to provide a history of*th^ ^ 
development and evaluation oi a set of training materials* Tlie goal 
of this upit is.tofhelp sjjHooI staff develop an understa^nding of^the 
Knowledge and skills necessary to identif y^ coileofc, organize, and 
^ analyze information jasefuL in asking decisions about program 

modificatiori. The "uni-^deals 'with the followi*ng topics: |1) ^ , 
identilying decis:^ons ^p^e made aboujt program ^bdifice^tion, and 
specif yL*g who makes these decisions;" (2) identifying alternativ^e 
courses of action for decisions in the instructional .man^gem^nt area; 
' (3) determining what ipfqrEration is Relevant to modification 
^decisions; (4) taking responsibility for or colitrifcutin^ to a plaii^ 
for program aodif ication information; , and (5) anaiyzi£g> and reporting 
information. Described in this ^report are the parts^ of the unit, the 
program**of development, testing, and field tests. A review of the^ 
unit /by two specialists- is summarized. (Authof/HLF) 
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» Wis ffepfert/is intended 'to provide a Ki^fory of ^Ijhe developnent and.^ 
evaluation of one of the products produced by the Educational ^Management 
Ihrogram. Reference is. made it\ the body of the fbport to mo^e detailed 
•reporti>/QT memoranda which dOcumenlt thfe dQveloprtent .and testing procedures 



used. Air of the^a^. documents are iiT^e^ program.' files, and many,ot them have 

been submitted to vai-i6us*fundiiig agencies that haA^e supported the prograiji 

. * ' . ^ * • . 

work. 

Tlie report has-been witten primarily for* thos^e associated with £he ^ 
National Institute of Education, and possibly som6 potential user^ of the 
product, who need to*make judgments about product quality but don't have the 
time to become familiar, detail with the" product^ itself and the extensive 
rpport6. prepared' during the development^ and evaluation of the product. ^^Jgie 
history of developnent i5 described in sufficient detail to suggest the 
aiijoupt of tli^ought, care, ^d discipline that went into creation- of-tbe 
product V Evaluation information obtained during developnent is presented at 
a 'level th^t would accurately reflect the strengths and weaknesses of the 
linit, .There is no separate detailed technical evaluation report ,|or this 
unit, but all of the data and documentation arfe available in the program 
'files. - ■ ' • • ^ H ^p ^ ^ - 

■ , ' .' ' ' 'Richard IJi.' Watkins 

^ * , • ^Program Diteetor 

' , • . * IMueational Management Program 
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11\e '•Development ^and Evaluation'^of • *- * ^ 

Evaluation for Program In)provanent ' 

.Evaluation f^^pr Prograifi Improvement 13 one o£ sqx sets o£ training 
matexidls' developed to increase the skilf, loiowledg^ <and understanding of 
^d\Qdl persorifiel in the area p£ instructional planning arid management In- 
s^AicUonal planning is 4ef ij^ed generally as that arela o£ educational admiii- 
istration that is concerned Svith providing edueatioTTal^ Opportunities for 
.children., Hus arqa may'. W further defined as jbivolving the establishment ^ 



rof instructional program purposes, the desiga and ijnplementation/t)f programs, 

^ ^ ^ • / . , ^ ' ^ ' 7 ' 

and" the evjiluation of instructional programs. The materials in Evaluatiq^ 

for' Program Improvement vire d^irected to the seconSsof these ^n[ee 'functions. 

Intended Users ' . 

^ These materials are designed for those school personnel who. are directly 

responsible/ for deciding Whether, and v^hat, pa^ts o£^ instructional prog^^ 

:,hould Be^^tered to increase the usefulness or effectiveness of the -pi;ograin. 

The specific role titles of such staff will vary from district to district;-. 
• ' - • * ,\ 

•but most <of ten will include building principals, d'epartment heads, district 

curri^alum specialists, and teachers^ wlio'are sending on currioilum coirmittees 

Depending on the size and organization' of a scRooi district or system, the 

unit may also be found useful by district office administrators other than 

those with direct cufViculum. responsibilities, Board Df Education members, 

'parents and students. It. should be clear .that th? intended user group can 

V ' ' . ; ■ \- ' • , . 

best be defined By function and iiiterest. ra^ther than by a> particular rple or 
title, and. thai; the group could include staff ^i)ot , necessarily falling und^r ^ 
the heading of administratiprs * . • ' . ^ . . 



Product Ilirposes and Description " . 

T!ie goal of "this unit is to help school *staf£ develop an* uncterstanding 

' • I . ^ . " . ' ^ ^ • • 

"o.f the knowledge and skills necesi?ary to identify, collect, organi*ze and ■ 

'aiiaXyzc information useful in making decisions about program modification. 
Tlie linit.'deals .with the following^ topics: ; . ^ . 

!• Identifying decisions that are both necessary and feasible to be 
made about program modificatior^, cUid specifying who makes these 
*. decision^ ; 4. ^ ^ * " » • • 

2* *Identi;^ing alternative courses of action for decisions in the ^ 

instructional management area; > . *^ • ^ ^ 

5* Determining" what infonteUion is relevant to modification 

decisibns; * * *^ 

4-/ Taking responsibility for or contributing to a pl^ for, ^ ^ 

/collecting, organizin'g and ^analyzing program .modification / 

\ 

information; and * \ ^ " • ^ 

* . - - \ " / * 

5. Analyzing and reporting in:^ormation. 

'Phis unit is designed to be use^in five sessions of three hours each • 
although some variations are possible to reduce the time required. Much of 
the. unit is based on an actual evaluation project and is presented in \hb . 
form of simulation exercises involving role playing • Session I, **Introduct;' 
to l^ogram Evaluation,^' presents an overview of evaluation in general and 
program evaluation in particular. It introduces a five-step evali^tion 
process ajlong with activities for leaniihg hc^to'use this ^process. Audio-, 
•visual materials,, exercises and writlien material are used -to promote partici 
{iant interest, involvement and discussion. 

In Sessions II,. Ill, and IV, the participants are divided into teams of 
six to eleven people and askai, through role playing, to simulate the 



/ 



activities *of aiyevaiuation coirimittee that is faced jwith the ^problems en; 

countered in the actual, evaluation, jStoject.. Aftgr each ^simulation exercise, 

the par.ticipan^is discuss what occurred in the meeting o£ .their "evaluation 

team'^and then review a description of what* took place in the actual evaiua-^* 

tloft ^)r&ject at ^his stag6» • . ^ . ' . , ' ,Jf 

^ ' In 'Session LI, participants identify decisions / decision makers and 

' - , ' * ^ * ' . . • • ♦ . • i 

olteniative courses of action tor , the simulated evaluation prcjett. \ ^ 

Session. Ill continues the simulation exercise/ .j^articipants identify Wha\ 
* • . , ^ . ' \^ *^ ' . ^ 

information the decisiQii maker will ne^ in decMoig on suitable courses of 

' ✓ . ^\ 

/ * . ' ' ^ 

actJfon and suggesting What insttuments could be iised to gather this informa- 

» ^ * .*> * • 

.tion. In Session IV, participants plan for th^ cblleption, organization and^ 
^ cjnalysis of infoimation, and consider how to dray conclusions from the \^ 
infonrfetion and make use'ful recommendations to decision makers. In the final 
session, participants draw on their experiencj? in the sijmilations to analyze 

/ • : ••• ■ • \ 

and criticize the actual evaluation design and final report. , 

- ' ** 

Reading material to be covered between sessions is included as prepara- 
tion for the next session^* Much of this material is drawn from the reporm 
on the^ actual valuation study, although portions of other publications that 
are particularly relevant are also used. An annotated bibliography is ijicluded 
in the materials for each session for "those^ who may, wish to .pursue particular 

^ ' •> M 

\ 

evaluation issues^ in.mojQp depth than impossible in the time available in the . 
training workshop.*, JJinally, the unit includes a fairly extensive bi^iog^^aphy 
organized in four categories : other evaluatiito training materials ; general ^ 
references on ^valuation; sources for mcasuran^t instruments; and general 
references on ^ucational measurement. ^ ^ ' * . ^ . 

Use of the unit r^ires a coordinator who has quitff-an active rol^as 
organizer and, to a lesser extent, as discussion leader. There is a separate 
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Coordinator Manuul ^ \v^l ps transpar^cies and a filmstrip and cassette tape:* 
Success ful use of the unit may^'reiiuire two or three days for the cboVdinator. to^^ 

amiliar with the materials prior to Yifst use* Hie codrdinator nbed 
not be an evaluation or measurement specialist, but it v^ould be^most helpful if 
the material ,in the annotated bibliography has teen read prior to ,the workshop. 
^ If the coordinator does not accept the decision orien^ted approach to evaluation 
underlying the unit, it may interfere with the success of the workshop. 

Development and X^sting • * • ^ 

The Educitiopal Management 'Program evolved fran an earlier program of 

the Laboratory Qoncemed with creating ai\ awareness and greater utilization 

by school\staffs* of the results of research and development effort. As work 

on this earlier program was being carried out, the need to assist school 

staffs in clarifying their goals and 'objectives became increasingly clear. 

llie developnent of a training unit on goal setting, .objectives and evaluation 

was first identified in the proposed scope of work to be completed in 1970 (Far 

IVest Laboratory, September 1969, pp. 80 ff.)* As the plans for this unit were 

worked out in greater detail, it became evident that more^than a single unit 

v/Quld be necessary. Prototypes and field test forms of two units. Deriving 

Goals and Analyzing Problems were, developed in 1970. The plans for a third unit, 

iJbje c^tives and Evaluation were described in general terms to be included in the 

.scope of woKk for 1971 (Far West Laboratory, September 1970, ppV50 ff.). During 

the summer and fall, hqwever, Banathy ^and Jenks were' completing the conceptual- 

ization of an instructional planning and management system. At this conceptual 
» * • ^ 

level, separate training units, •one on deriving objectives and one on program 
. * , , ^ 

evaljiat ion /were* identified^ The former fell under the general heading of 
rogram purposing, "and included planning* for program monitoring; the latter 
fell under the hea^g of program management, and included both evaJLuation for 



■program modification ajid evaluation for final d'ecision makifng* > 

* . / ^ ' ^ ^ • . . 

» .First unit prototype / The development of a separate unit on derivin'g 
» ^* ♦ • ' . . • 

.objectives was initiated in J^uary 1971 j .planning, for prograirf evaluation In 
K. " »« ; - // " . , • » " 

TTelatiori to the objectives was not included witliin the definition ''of the unit^ 

9bjecti,ves. Concoptu^liSation of, and detailed planning for a separate ufiit 

on projgram evaluation was started in the, spring, of* 1971 and presented in ^ 

detail in a staff papei: (Harris, Aug^t lp7l). . *The initial plans for the 

lQ]it were deriyed from a systans analysis approach, to evaluation in, which the 

— •> 

i . . * 

evaluation subsystem consisted of four separate functions: defining injiica- 

tors of objectives and 'setting standards; collecting. data; analyzing data;. 
Aid identifying necessary program adjus,tments. Very detailed behavioral 
objectives for each/of these funct^ions, or elanents^ of the evaluation sub- 
system, were stated, and. ihe obj^ectives for the unit ^eve in effect to enable 
trainees to achieve these functional .objectives. Fi"^e Idnds of program 
evaluation were also identified: design; ^unplementation; process; outcomes; ) 
,and costs. - * , ' * 

Two major^unit orgaijizations were*anaiyzed in detail, one using' the^ 
functional approach and the other using, the program evaluation types approach. 
The functional approach was seen to result in a -shorter unit that, if success- 
ful, would imi^art more^ general izable skills and laiowledge. It was, however, ^ 
decided to organi:;e modules around me types of evaluation even though it would 
require much greater effort. There were three reasons for this choice: 

1. It was more nearly consistent with approaches used by others in .the 
prepara:tion Of evaluation training materials and particularly con- 
.sis tent with the approach being followed by the Center for the Study 
of 'Evaluation at UCLA, This meant tliat the new unit could make .use 
of existing materials, oT could refer useys to existing programs^, 

• rather than developing an entirely new unit. «^ 

. A . 

2. The skills and • knowledge would be more immediately applicable to the 
' solution; of school 'evaluation problems. 
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3. The unit' could be developed in a irtbdtr^r fasMon so that u^ers CQuld 
.choose training for a particular type o£ evaluation that they .per- 
ceived to be mo^t relevant to their problems. 

Thfe plans were put into effect^ and a prototype version'^of the unit com- 
^pleted (Far West Laboratory, Septanber 1971, pp. A 48-49;, Harris, October 1971), 
Hie prototype unit was composed of an Introduclfe^ry Overview'module, and five 
additional modules, each^ directed to one of the five types of evaluation. Be- 
Cause of the modular "approach, the unit materials were very voluminous and com- 
plex. Planning for .the evaluatioji of an actual externally developed high schqpl 
political studies program in a school district by a committee of school staff 
was usefl as the ba^^for the unit. There was a considerable amount of pro- 
gi'ammed learning, used in the unit; but the burden of the training was commOni- 
cated through lengthy comments by an external evaluation consultant working with 
the wommittee, which were to be read by the trainees during the training sessions 
A systems analysis approach to evaluation was stressed, and attention given to 
the possible uses, similarities and differences of various models of evaluation. 

The overview and each of the five independent modules w^re distributed 

for review to Laboratory staff and selected practicing school personnel. 

Generally, the Laboratory staff mmbers considered tlie content of tlie modules 

to be quitu good, but had some resen^ations about the format and wordiness 

of the presentation. The external school reviewers judged portions of the ' 

various modules to be quite useful. They were, however, rather negative about 

the fom and length of presentation and quite critical about the general 

didactic tcme of mucn of the material. They felt that some parts of it were 

insulting and^that the structure of the modules forced tljan to spend a large 

amoiint of time going through material . they already knew in order to learn the ^ 

new material. They also, questioned tlie usefulness to sch.ool staffs of some 

* ** • • 

^ . 

of the content of* the unit. 
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Unit replanning ^ •At the^^ime the prototype test was being cannpleted,. 



thfere were several changes in personnel unrelated to the development o£ the 
unit. In view of the overall reactions of the school baseS reviewers of the 
unit, the decision* was made to undertake a replanning effort that would-be 
based on field based interviews with school stafjf . The interviews were to 
include those v^io had^ responsibilities specifically for evaluation activities, 

V 

building principals, teachers and curriculum specialists. An initial inter- 
View study 'Was conducted by a staff member traiijed in sociology and was 
carried out in two school dist^'icts: Oakland and Novate, California. The 
former district is a large urban district that had a number- of ESEA Title I 
projects with ^ heavy emphasis 'on evaluation; the latter is a suburban 
district .that had several innovative projects under way in the schools. ^ The 
information gained, from these interviews was pres.ented in a staff paper 



(Cassells) . 



Following this initial effort, a number of evaluation specialists were 
asked to nominate exemplary evaluation projects they knew Jiad beei/ conducted. 
The staffs of these projects were contacted and many were subsequently inter- 
viewed.^ The purpose of , these interviews and obseiv^tions was to learn \yho did 



what, and how, in these exemplary projects and also to leam what, the various 

V - 

participants in the projects would like to have ddne differently. . It should 
be eniphasized tliat these various interview studies were not conducted in any 
kind of systematic my. They were rather hastily arranged efforts to. gain a 

■ ; ' ■ . \ . 

better understanding of the self -perceived needs of school staff, for prepara- 

» , » 

tion that would help tliem in collecting and using information about sthool, 

programs so that a revised evaluation unit could be planned to respond to 

* 

these needs- ' - i 

/ * ' r 

The infonnation obtained in this fashion was summarized in a planning 
paper that, also set foi;th the purposes arivl general outlines for a unit That 
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would most likely be seen as helpful by school staffs (Educational Nlanaganent 
Program, June 1973)/ \Tne proposed unit had two major, goals, with sets of 
, .goal *,^icatorst; and with.eyen more detailed objectives derived from the goal 
* ■ indicatory The first major gbal^ might be seeit as dealing with wha\ is 'often 
/i^alled formative evaluation, knd^the second goal- seen as dealing with the 

• '* * ♦ " », , \ ' 

- general area of summative evaluation^ The idea of collecting, information 

' < ■ • ; "I- V* " " > ' ' 

. that would be•^seful to people' who had to make program decisions of various 

kinds Underlay the general plans. for the unit at this time. 

Initial work on *'\^s reconceptifel^zed unit was started, recognizing that 

the two major goals of the unit and the related objectives were highly inter- 

. . active^ That is^l^while the^ goals an^ objectives for a imit might be separable 
•* . , ■ 

•» into fotinkti\^e and sumnrati^ evaluation, in practice the activities and accom- 

. plishments of school^staff that contributed to these two kihds of evaluation 

^were not, and the unit ^obably cqyld not be neatly divided in this way. 

- • * ^> ^ • • ' " . ,i J 

* * Second unit T3rototypfi r^,J)ra£ts of. the material for four three hour, 

' * , ' J. . ■ . ^ , 

training sessions w^re completed, drawinjg heavily on infomation and experience 

gained- in an adtual :^chcx)l* evaluation pro j opt where, the author qf the draft 

had been invol^ved as an. external evaluation consult&it. Insofar as possible, 

the pli^ing team approach v/as^sed in the^unit, but with a strong emphasis on 

' , ^ ^ • - ^ * * ^ • . \ •/ . 

^^^active trainee involvement through rotfe. playing as members of the evaluation 

team. . ^ ' ■ ; - . ,■ 

These drafts vyere used in four class sessions of a graduate course in 
' * - ' ' ^ * \ , ' , - ' ^ 

Gjrr^tulum^, Evaluation bein^ conducted at California State University, San 

' Frarfcisco (CSUSF). Virtually all the clas^ monbers were practicing teachers, 

93unselorvS or- administrators. Program st-ajflnem^ers served as observers of 

these sessions, and^itten questionnaires and conments were obtained from 

the' participants. Two major conclusions were reached as a resuJLt of this 
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first trial: First, while .several of the role playing sessions worked 



moderate.ly vvell, suucesb Jepended-^very markedly oi^'tlie fortuitous cfomposition 
of the role playing groups, cind once the role paying and simulation broka 
.down it w^s difficult to get things going again. Second, and even more 
important, the participants did not seen able to deal with the idea of 
decision making in any systematic fashion, nor was there any common acceptance^ 
of the idea of program evaluation as oppdsed to teacher or student evaluation. 

Three independent analyses showed tnat approximately half o£"the objec- 
tives identified in the planning paper had been covered by this first draft o^ 
material. At about this time (Novanber 1973) the staff was informed of the 

planned termination of the program in one year, with greatly^ reduced funding. 

• / ' *■* 4 

/ * ,2 

The decision was made, therefore, to focus as much as possible on the formative ^ 
evaluation goal of the unit. • ' 

First field test version . Following this prototype te^t, material for a 
completely new first session was prepared and tested separately in a class at 
CSUSF. This material was designed to introduce participants to the ideas of de- 
cisions, decision makers, contributors to decisions and factors that affect 
decisions. The first draft appeared to* be usefid but also in need of further 
revision'. , ^ • . 

Thb materials for what were now five proposed sessions were extensively 
revised and edited, and supplementary readings for each session chosen and an- 

notated/ As soon as.^the revised materials were qompleted in draft form, seven 

^'"^ »^ 
people who had had a variety of school staff experience were asked^tOv^o through 

/ 

the unit in five successive half-day sessions in July 1974. A program staft* 

/ ' 

member served as coprdinator, and at the end of each half-day session, the par- 
ticipants offered detailed and useful criticfues p£ thfe material. Many of their 
suggestions were incorporated in the materials used in the next field tests. 



Arrangements for two field test sites to be used in August of 1974 were 
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made.^ One of these -^as a one week experimental simmer school graduate .course' 

at CSUSF. Hie other was a four day workshop sponsored as part of the Pro'- 

fessional Development Program (PDP) 6f the .^sociation of California School 

:-*\dministrators (ACSAJ, and held at Marymount College in Palos Verdes, Cali-fomia. 

Tlie experimental summer school coiirse was one of two offered at CSUSF 

during the same week, one in the. morning and one in the afternoon. Only four 

people registered for the evaluation course in the afternoon. Having been- in 

the course on program design in the.nioming, they were fatigued and relatively 

unmotivated by. the time they came to the afternoon session. By and large, the 

field test was not successful when judged as a field test,' primarily- because ' 

of these administrative considerations'.^' The staff had, however, contracted 

' ' ' \ 

tor a non-staft observer to attend the five sfessions and to assess the use- 

» 

fulness of the materials for each of the 'five sessions. Subject to the 
limitations \if the field test situaiti*ons, his written interlinear suggestions ' 
and report were quite useful in subsequent revisions. Three of the four 
participants judged the Workshop to be "fair" and one judged it to be "good." 
None described it as "excellent" or "poor," but free response, coirarients were 
generally negative. 

The same version of the materials was used, in the ACSA' sponsored workshop, 
but since only twelve hours were available, the coordinator had to combine por- 
tions of the third and fourth sessions into a single session. The time limita- 
tion prevented collection of detailed information about effect's of unit' use 

\ 

but participants were asked to give overall reactions and opinions about 
strengths and weaknesses of each session. Additionally, a program staff member 
obberved the entire workshop to identify potential problem areas. Tlio workshop 
coordinator was a Laboratory s^taff member not previously familiar with the unit. 
The workshop participants were school staff members who chose this 
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particular training from among five different ''courses" offered by the 

I'rotessional Development Programs. Twenty people completed the wor4;shop. 

was a teacher, nine were principals, four v;ere district superintendents 

t> . * ^ 

.and six classified themselves -as ''Other Administrative Positions." 

Fifty percent of the participants described th^ workshop as ''excellent*' 
•ind forty-five percent described it as "good." One person described as 

I^poor" and ^aid he thought the use of role playing was a ver>' poor device. 
i>f those uho had had no e^cperience in evaluation, two-thirds judged the 
workshop as "excellent," and the other third judged it to be "goad." Forty- 

*fi\;e percent said the material was "moderately diffiaalt," and forty- five 
percent said>it was "moderately easy." 'I\vo people said they thought it was 
"too easy." Tliose with no experience in evaluation, rated the unit as somewhat 
more difficult than did those who had had experience-. 

At the end of the u'orkshop, partici^aJ^ts were asked if they would be 
interested in conducting a field test of the unit in their district.' Six of 
theiu responded positively. In foHow-up contacts, only one supgrihjt^eijt was 
able to<^rran^e for staff participation in a figld test. He recommenced thatSs^. 
county office sponsor .the field test so that some of his staff could participate, 
When arrangements u ere made, the superintendent provided released time for four ' 
department hjsads and a curriculum special ist/iand^'^paid their tpvel and lodging 
expenses for a two and a half day unrkshop.Arae other five who hjid indicated 
interest in field testing were unable to schedule. field tests within the limits V 
imposed by the program contract schedule/\but several also express ei^ re^iratioij^ ! 
about »their own preparation as coordinators, \ » v. \. • j 

St'cond field test versit)n. The experience, infonnation i^rom p4rticipants '/ 
and staff observations from these firsh: tu^o workshops wer^ used, in 'revision ./ v 
of portions of the materials. Most of the rel^isions were made m the first 
ses.sion deaMng with the issues of decisiohs an^ decision makers. Additionally*, 

le 



attjbntion wab given to plans for the collection of better summative evaluation 
information than 'had been collected previously. ' * . '\ ■ ' 

Four field test sites were arr^ged> three^ in California school ^ 

r i ' 

da.gtricts and one in the Alaska State Department of Education, A total* of 
64 participants used all or parts of the materials. Approximately one- third 
wei^e teachers and a little more than half were principals or other adminis- 
.trators; five vyere school counselors. ^ •* " . • 

Thirty-nine of these participants responded to overall judgment questions 

^ * " ' f 

at the conclusion of tlie workshop. Of these, eighty percent said«that they.h^d 

leam^Hi what they had expected from the workshop, ten perceilt said tliey had not 

gotten what they expected, and ten pejcent said they had mixed reactions. The 

participants were also asked what they would recoirenend to a friend if he sought 

advice about participating in a similar workshop. Eighty- five percent indicated 

that the unit was worth the investment of their own time. ^Hie results are . ; 

presented in more detail. 

l^esponse 

^ Take on own time, with fee of $10 
^ ^ Take on own time, if no fee 

Take on released time, if no fee 
Do not take 

In order to assess effects of the workshop, it was decided to construct a 

set of items tHat would appear to solicit opinions about evaluation procedures 

or values. The^ item content was chosen, however, by a detailed consideration of 

the purposes and content of each session in the unit; a five-choice ''agree- ^ 

disagree'' statement was written if in the judgment of twa of the developers' ai^ 

expectation of a parjticular response could be based on the unit content. Thus, 

while the items appear to assess. oi)inions, they-migfit be judged to have content 

'•. * * 

validity as indicators of unit effects.. - ' <^ 



Number 


• Percent 


21 


54 


,12 


' . 31 


4 ' 


10 ' 


2 


5 



Approxajnately fifty such itfems were written, and fo.ur Laboratory Staff 
members, knowledgeable 'about evaluation, were asked to respond to them 
without knowledge of the content of the' unit. Some itams^Were edited, re- 
witten or eliminated. Care was also taken to insure that the "correct" 
responses were about evenly divided between "agree'? and "disagree." TVo 
forms of the opinion questi&ns were constructed, judged to be roughly 
parallel, ^ The?e forms are included in the Appendix A, with the "correct" 
answers circled. / . 

; ' . \ • 

It seems liKely that these two, forms would yx^ld good scales of opinion 
and might even yield a reasonably good Guttman scale. Lack of time alid - 
. funds 'have,, however, prevented any'test of the scales. Scores were obtained 
.on a scale from zero to\99. In three of the four sites, randan halves of 
the participants were asked to complete Form A or B before the workshop aiA 
the alternate form after the worlcshop . Analysis of these d^lta showed that 
the Forms A and B were equal in "difficulty" with means of 79 and 78 and 
standard, deviations of 9.3 and 9.8. ^ ' - 

The relationship between the pre- ^d post-opinion scales, pl)oling 
data from the two fomi^, for the 32 participants who provided ^usable data ^ 
are shown in the t^ble beloWj. It should be noted that many more than these 
32 completed the workshop. A number of the -participants did not notice 
tha-t the opinion questions were printed on* two sides of the^same sheet and 
so did notTanswer all items. In other instances, the participants' work 
schedules did not pem.lt them to stay proMde the posttest data. 



1 

\ 



14 - 



66-75 



Table 1 

Pr6- and Post-Workshop Opinion Scores 
* Post-Workshop Scores 
82-'89 



74-81 



in 
o 



90-99 
82-89 
74-81 
..66-75- 
58-65 
Totals 




Totals 
5 
7 
• 7 

: ,10 

-.• 5 

32 - 

: • . / 

■ „■• Table'2 

■ . 

Statistics^^ for Pre- and Post-Workshop ^^iflions 



J.; 



Pre^qpre 



Mean* 



^Jr Standard 
r "^^^^l^^vi^tion" 



'78 
9 



Post-Scor^e 
' 89 



Post - Pre 



11 



,-6 



11 



It" is ardent from these data that, as a group, participants did alter 
their expressed opinions, in the directions .indicated by the content, of the 
material. If the band indicated by the diagonal lines is used as indicative 
of no change J.n opinion, thW one can conclude that 23 (72%-) changed in a 
^positive direction, thi;^ (91) changed in a negative direction and six (191) . 
showed no change. The pert^tage of positive change seans qu^te marked when^ 
^y^^ considers the fact that the scores on the pretest were gui^e high before 
the workshop, i The mean change^is statistically significant.^"-^ 

In addition to information related to use of the unit tha^^^was collected 
J by the staff, threfe other kinds of evaluation evidence are available: an 
unsolicited repojt Mom a field test; actions of the ACSA Professional. 
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Development SrJ:aff ; and expert reviews of the second field test -ysersion. 



Uns*olicited fiej.d test report . The staff member fran the University of 

Alaska who had hancjl^d the arrangen\ents for the field test site in that state 

sent to the program an unsolicited report (Hecht). Shis had prepared five 

questions about the workshop that were of interest to her and she distributed 

the questions to' the participants- at the^ Conclusion of the workshop. Responses 

w^e to be mailed; nine of the twelve participants responded. The individual 

resp^ses are given in t^iF entirety in the report which is included as 
• . ^ 

Appendix B. The five^estions an^ a classification .of the resppises is f 

given her€\ the responses were classed /as positive, neutral, negative .o^ 

unceVtain by the author of this i^ort. * 

Dp you feel you haye learned anything in this workshop which 

be useful in your areS of responsibility? Explain, \ ^ * 

Eight of the responses^ere considered to be positive, without 
reservation; -one ivas (fonsidered to be* neutral or positive with 
^ some reservation. ■ . , . * 

^Vhat yc^uy opinion of the materials presented at tlie workshop? ^ 
a. content? b. format? " * . ^ 

Five responses were Considered to be clearly positive, and four n 
* to be .neutral or pc^^ive with reservations . ^^ > • :* ^ 

3. Do you see any . use for these msCteri&ls ^or similar, in yoi^r job 
situations? ^qplair^. - ^» ' ^ 



Four responses were considered positive, two positive with 
reservation, two were judged to be negative, and ^)ne coiild 
not*te classed. At least some of the reservations seood fo be 



; related ^^tq characteristics of the job situation rather than 
to^ the unit per se > * ■ ^ 

4. In your opinion, shouM AEPIC/CNER consider sponsor irig. 

similar workshops in the area of program evaluation? Briefly 



explain. \ 



, Five i;espoii^es appeared to be positive) three neutralyor 

I uncertain, and one negative. The responses sem to be directed 

^ to the general quefetioi\ of the value of workshops, and so only 

* an^ iiSiirect assessment cff this particular workshop. i 



5. Please comment freely as to any additional reactions, suggestions 
etc* you may have... 

Five responses were interpreted as being quite positive about the 
workshop, and one as being positiveT)ut witH reservations about 
future use. The others could not be judged, or related to iktters 
other, than^tliis particular workshop. 



Overall, thk report seems to be quite positive about the* materials and 

the workshops; but there seem to be some who have reservations or even 

negative feelings about the value of the materials to them' or about using 
<) "^^^ 

than in the future. * ^ 

ACSA-Professiofel Developnent Program actions. After the ACSA- sponsored 
Workshop in Augiist, Dr. Edward Beaubier, Assistant Executive Secretary of 
ACSAj^^O'eported that a number of the participants spoke to him very positively 
about l^e workshop. They recommended that it be ;included as a regular part 
of, the ACSA-PDP offering. In the faljl, after the three .district workshops 

f 

had been completed, the' liaison administrators for PD? did, in fact, vote to 

include it in their offerihgs. ACSA has now contracted with the Educational 

A. ^ 

Sefvices Division of the ^.aboratory to purchase materials and coordinator 
services to offer the workshop four times in different parts of the state.. 

Expert review of materj.als > After final revision of the unit materials^ 
arrangements were made to havp'^o specialists in evaluation review the 
materials and respond to ques^tians posed' by t^e program staff* The questions 
* are included in Appendix C, biit the reviewers' wer6 encouraged to offer any 
comments about the materiais\hey thought *in btder^. ^ * • 

The two reviewers were chosen to, represent two quite different kinds' of 
experience: They were: •» , ' . 



Dr. David PaynQ ^ • Mr. Reginald Corder 

School ot Education . * Evaluation *^d Advisory Services 

The University of Georgia Educational testing Service 

i ♦ o ^ Berkeley, 'California 



Dr. Payne has frequently taugJit courses in curriculum evaluation, is 
the author of several books on measurement and evaluation and has served as 
a consultant on school evaluation efforts an4 on the development of competency 
based., administrator trairiing. Mr. Corder has had extensive experience as a 
secondary school teacher and'^ counselor and, for the last ten years, has been 
extensively involved in providing professional consultation to^ school 

plaiming and/or conducting evaluation projects. i. 

Following are quotations. from Dr. Payne's letter of transmittal sent 

with his report. ^ 

''After studying the materials from cover to cover, I must conclude 
that they ire the best I have seen. .The paclcage is far superior 
to the Worthen et al. simulations. . .and of course your materials 
are only a fraction of the Worthen costs. Qn^ index of my enthusiasm 
is m>^ intense desire to use your materials ^this summer when I teach 
a course in curriculum evali^tion, • .several colleagues of mine' from 
one of our state Cooperative Education Service Agencies may be con- 
tacting you... about the possibility of securing the materials for 
use in a workshop in March." ' 

Excerpts from Dr. Payne *s responses to the questions posed by the program 

staff are presented below; the complete responses are given in Appendix 

1. Adequacy and Responsiveness of Unit Goals and (^jectives . 

"This reviev/er definitely feels that the evaluation unit would be ^ 
appropriate for principals and instructional supervisors, and to. a 
lesser extent for teachers. The goals listed on pp. tj;-a?, and ob- 
jectives identified on pages 1.3, 2.3, 3.3, 4.3, and 5.3 are definitely 
inportant ones, and if met and when applied could result in improved 
instructional, programs. One wonders if objectives dealing with tlie 
mechanics qf undertaking cost- effectiveness analyses have been 
slighted. Tlie five steps idontified in the evaluation process are 
. logical and the ones usually stressed in such training programs." 

2". ^' Relation of Unit 'Activities to Goals and Objectives . 

"In general .activities are highly responsive to the session ^« . 
objectives. One possible exception might be tlie decree to which 
' Obj.ective 2 on p. 2.3 and Objective 2 on p/ 3.3 are treated 

adequately in, their respective sessions. ' ^ ^ ' : ' 

* * * 

Obviously great care has been tiken In the ^ development o££he 
materials. The directions for the coordinator are the most complete ^ 
this reviewer has seen. The material on role playing and discussion- 
leading is most helpful. In addition' the explicit and implicit 
emphasis on decision-making is to be^camiended." ^ 

22. • * 



3. Tasks of the Coordinator > . , ^ • 

•'Obviously some background in research, measurement, and Evalu- 
ation, both acadmic and experiential, would be most helpful* to a 
potential c(X)rd-inator, In addition some human relation training 
would staiid a coordinator m good, st^ead," • 

4. Training Tiine Devoted to Unit >and Sections > 

"In general tasks seen) relatively well balanced for time. This 
reviewer would be .very hard pressed to eliminate any of the sessions, 
Tlie notion^ as suggested* on p. w of using a l2h or IZh hour 
schedules does not seem like a viable ^alternative. Tlie saving ^of 
time is not that great, and yet considerable loss in training over 
important topics would probably result^ ^ 

Despite tiie fact that they have been field tested this, reviewelr , 
finds it difficult to accept the fact that Sessioiis 2 and 3 each 
really require three Ijours.'' 

5. Cost of Materials . \ 

"There is no doubt that the projected pricing of the unit will 
be one of the strong points in it's appeal. Most, other similar units, 
particularly those available from coiiinerdLal sources, are priced 
three or four times tfie present price. Costs are well witfiin the 
budgets of virtually all public schools and those of graduate students." 

t 

* • * * 

6. Use of Unit, ^ 

"This reviewer would have no reservations about xising the unit 
either in a college *-or university teaching situation or in conducting 
, workshops with public school personnel pr professional organizations 
(Assuming that they were relatively inexperienced in evaluation)..'/ 

''.•othe Ciiief "competitor" with, the Far West materials , ' 
^ would pi'obably be "the simulation exercises ^developed by Blain 
Worthen. These materials run into several hundreds of dollars , 
and are not nearly so detailed in presentation, particularly with 
regard to what participants and coordinator are to do, as are 
the Far West materials. The Far .West materials are far superior 
with regard to 'objectives . and sped^icatioh of activities, ' 
, responsibilities, and roles of both participants^ and. coordinator." 

Excerpts from Mr. Corder's report, whidi was in the form of, a letter are 

quoted below. The entire letter is included in Appendix C, except for the 

initial and final paragraphs, v^^lich are no't related to -this unit.^ 

''I was pleased to see a training package directed to program 
improvement, and to an audience-at ^the building level.. .^fore and 
# more I have cme t6^ believe tlia^^p^program improvement and 
. actual use of .evaluation data th^^^rsons actually engaged in 



the program, should have major responsibility for planning 
conducting the evaluation. . 

/ * U 

^ '"1 would like to lise the ma'terials in a situation where 

building staff wish to evaluate a new program. • .iriost small ^ 
projects, at least-, and building level projects think of the 
evaluation too late", not at the time of program' development* 
I iyould be highly receptive to a' call from a superintendent 
which went, 'Come and help us plan for the evaluation of a^ 
program which we .expect to get under way next fall (or spring 
or, whatever) . ' The opportunity to use your materials under 
these circumstances, to get program staff involved in the 
elements of planning for program improvment, and then later 
^ • engaged in worlcing on the evaluation of their real program would 
^ " be attractive to me. I am most hopeful for the materials under 

these circumstances., IMle such circumstances are ideal, I also ^ . 

think that the materials would be successful in a less ideal s 
situation where I would say, /'Okay, I'll cane and put together . 
an evaluation which wii; sav6 your project funding now, but as the 
next step you must -provide staff tme foi* a workshop in whidi they^ 
will leapi scmething about planning their own.evaluationo* 
•».Even \inder forced conditions, however, the materials ^re pre- 
sented in siimalations which* could engage tKe interest of a 
building level ijudience." 

C oncl usions ' * . . 

— ^ - . ^ ' V 

The 'information* that has been obtained about t^^is unit indicates that . 
^ * 

it is effective as material for training. It is generally seen as useful by 
those -v/ho have used the material, .ani meets a need that is seen as important 
by both, school personnel and noij-school 'specialists. The major unanswered 
questions are: the extent to which the unit is dependent oh the personality, 
qualifications, and evaluation philosophy of the coordinator; the amount of 



preparation required by coordinators who may vary 'greatly in the qualifications 
and the variations from on6 situation to another in the usefulness of the role 



playing • In one instance^ the need to assume a particdlar role in front of 

colleagues in a district appeared to create a conflict and perhaps even a 

« . . - • 

personal problem for one participant. ^ . 

* • ' / 
\Vhile it would undoubtedly be good to devot^6 additional effort to getting 

more information about the use of the i^it , fii^ds ^re not available for such 
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■anomdertaking. On balance, .the available irifomatioih seems to justify i„dking 
the unit^in its present fpm available for use* in sdiools and universities. 

The unit Jias been reproduced in limited quantity, and ig -available for 

I ' ' ' ■ • ■ ' 

sale through the Laboratory. The Participant's lian^book i-s priced at $7.95; 

the Coordinator's Handbook , transparencies, and' filmstrip and cassette tape 
cost $34.95. ) • " 4 
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Date: 
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append/x a 



Name: 



OPINION pUCSTIONHAIRC 
w {Form A) ^ 



Indicate whether-:^ou agree or disagree with the followiiig statements by 
'circlin^j *onc of the numbers f rotn one (strongly .disagree) to five (strong,iy 
* agree). Example: "Charlie Brown is a good baseball manager." 1 (?) 3 1 5 
^ ^ ^ • ■ • * disagree * igil?e__ 



. 1. Bvalut;tlQn should fae perforV.sd only when a prograrans 
. • completely developed. * ' . 

, PT Evaluation studies sTiould determ'ine the correct' decisions 
* ^ about how to change or improve a program* 

' a. The' real testvot an evaluation is v/heiher it helps some- 
-V, ^ one >iake a^ better decision on jibw to, improve a program. 

4. •Evalifetion sHould not be^done by people who are on thfe 
" schooVd-i§tr.ict staff., ' " - * 

5., Evaluation j's not a very con^nqn ljuman activity^ 

5. «»The first*step in planntngrsfor evaluation is ,to identify 

^the Dossjble decisiohs on hen*/ to improve, the program. " • 

7. """Terminate the prpgVam*," is a good example ^f the kind of*., 
decision Whicht could be listed asrpossible at the beginning 
of an evaltibtiori study. 




J 3 

* 

r 2/ 3 



4 5 

4 5 . 



o 



3 
3 



1.23 
1 2 3 



o 

4 5 



4^ 5 




8. Evaluati»n information will be most useful if^it is related 
, to partfcular dgcisions and deqd si on makers. '• 

Tnm2 arttJvmpney ar^c likely to be wasted if possible .alter,-^- 
'native actions dr -JmprovcRients in the program are specified" 
in advance^ . ^ . 



4 5J 



o 



1 z, 3 
345. 



o. 



10. All fedsrble ctlternatives should be corisiderod before 
{ ^ making a (incision. 

11. If alternatives- t'i*c not specified in advance., information . 
necessary formating the decision may not btj collected." 

12. The Vtilui^sr idOfils, .ip.'] c^qpstraints ti^ffocting decisions 
aro^amonrj the less important factprs to consider in plannin'g 

•* and C9iidi<cting"i|r\'='evaltiati-on. 



1 .2 3 
12 3 

O 2 



o 
o 

4 5. 
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Opinion Questionnaire 
Fonn A^, p. 2 



Name: 



disagree . agree 



'13'. Instruments should not be chosen until information neids 
and probable sources have been identified. 

1/k Methods to'be used in analy^ng the information that will 
be collected in an e^taluation-study do not need to be vj 
outlined until after data are collected. » 



1 Z 3 
3 



■ O 



2 3 



15. information needs can bo identified by matching program. 
^ 'characteristics with the 'consider^itions important to 

the decision maker(s). / 

* 16. When planning information collection, it is important to 

coas"^'der whether enough subjects are available to justify the 1 2 3 
cost of collecting and*anaiyzing particular kinds of data. 

17- Community publications are often a good source of infqnnation 
' for use in foiSriul a ting .questions for program evaluation. 

Standard itthl tests are a complete source of information 
on a program. , ^ , 

19. Even in a well-run^ evaluation, cons-ultant help or data 
processing services ""may be necessary* 

7SS. The interests aad requirements 9f the decision maker must 
bfe a prime consideration in how' the evaluation report is 
put together. 

21. Simplicity is irrelevant in data display. 

22. The evaluator should plan how to obtain evidence about the 
relevance and reliability of data to be collected in an 
evaluation study. • , " » 



o 



4 5, 



o 
o 




1 2 3 



NOTE: Circled responses represent '-'correct" answers as determined from 
unit content. ) ^ * ^ . 

« - *• 

'Scores represent the total of item values, obtained as follows: 



Response 

"Correct" 
Neutral 
"Incorrect" 



R esponse Value 
4.5 
0.0 



o 
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Name : 



OPINION QUnsriONNAIRE 
^. * (l-onn B ) 

IndiLvito whetfior you agree or disagree wiih the fallowing statements by circling 
one of the numbers from one (strongly disagree)^to five (strongly acjree). 



example: "Snoopy should got the- Re^ Baron./* 



1 2 3 4 (5) 
disagree agree 



Evaluation should be performed while the program is being 
developed. 

Evaluation studies should be used in deciding whether to 
retain, or drop part of th€ cui^riculura. 

Teachers should have a major role in planning'and con- 
ducting evaluation studies. 



:5/ The rtal test of an evaluation stucly is whether or not the 
evaluator is satisfied that he knows how good the program is, 

6^ In planning for evaluation activitips, it is not important 'to 
di^inguish information gathering decisions from program 
modification decisions. 

"Cljange the goals and objectives," is a good example of the 
kind of decisions which could be identified at the beginning 
of evaluation' planning. 

8. It is compara^tiycly unimportant to know who has the authority 
to make decisions about program modification. "\ 

9. Specifying potential alternative improvements iti advance 
helps one' focus ^valuati oh activities efficiently. , 

10. AM feasible alternatives should be considered before 
makifig a decision. 

li ' 

11. Possible decisions and alternatives cannot be identified 
until the evaluator has specified necessary inlormalion . 

. • i - ' . 

12. Proijram cos'ts, academic effectiveness, and student 
attitudes a^o usually relevant considerations in deciding 
hov/ to modify a^ profjr.im. • " 




1 2 



o 
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3.4 5 



1 2 3 



o 



1 2 3 

3 4 5 

f 

1 2 3 



o 
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^9 



(i;)iin'(>n Qu'-'il iontiaiiV 



r 25 



^(11110 : 



disa(iroo 



a (J roc 



13. IristnhVMil', ,shouU! nol bo cho*>(-n u\\U^ irifonnati.on ricocir. 
nnci pnlvlilc zokitl^o oi inforiiiatioii have betni identified. 

14. If infoM'kiUon .6nolyi>is i:> planned ahead, information may 
he coIIocLlmI in an incvonveniont form, 

15. Inforn: lion needs c.)n be identifed by matching proqram 
charcK:t^nM>>ticG with the considoraLions important to the 
decision n;aker(s); 



1 2 3 fl 




1 2 3 



^ 5 



o 



f 

IG. It is essential to schodulo data colloction activities" and 
♦ personnel assigniicnLs before iii-plomentiny the evaluati6n plan, l 2,3 

17. Info»!nation about students can be gotten only from them. 

18. Cvaludtors should develop all their ovm instruments. 

15. A well-trained evalruitor should be able to handle all 
aspects of data collection and processing. 

20. CoinprehensibiTity is a prime consideration in data display. 

21. In order to be useful at all,, evaluation recommendations 
must be written with specific jlecision makers in mind. 

22. The evaluator should p'(an how to obtain evidence about 
the relevance and reliability of data to be collected in 
an evaluation study. 




NOTE: Circled responses represent ''correct** answers as detemined from 
unit content. ^ 

Scores represent the total of item values, obtained as follows: 



Response 

**Correct'' 
Neutral 
**Incorrect'* 



Response Value 

. ^ 4.5 
' 2.0 . 
0.0 
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APPENDIX B 



■ Report on 
AEPIC/CNER Sponsored , 
Evaluation Workshop , 



-October 14-16, 1974 



••JKafchryn A> Heehfc \ y 
AEPIC Evaluafcor 
Center for Northern Educational Researcli 
University of Alaska 

November 25, 1974 
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On October 14-16, at .the Holiday Inn in Anchorage, AEPIC/CNER 
coordinated an evaluation workshop for twelve participants from a 
variety of Alaskan educational .msjtitutions (see list of participants 
attached), cj;^ ,^ ^ • ' ' - J 

The materials used dirr^g the workshop werV supplied by'the 
Far West Laboratory "for Educatidh^l flesearch and Development, 

This workshop is entitled '|Evaluation^for Program Improvement" ^ . 

^ \ ^ \ ■ * 

from a training series called "Designiflg Instructional Programs". 

The workshop leader, JDr, Earl Mortensen of £he Lab, was one of 



the developers of these training materials.'' . I 

As evaluator for AEPIC and workshop coordinator, Ldecide^ 
to let the participant speak for tiiemselves on the value of this 
and othe^ such activities in termil of their pr^fessi^ohal need, as 
they are certainly in the best pdsition to judge. Their answers to ^ 
my questions are rerfofted fully in the following pages, 'The ^ 
questions were discussed duJring ^ brief introductory talk the first 
day of the worl^liop'and passed out at its conripletion, to be returned 

by mail. 'Nine^of the 12- participants responded, as of November 22* 

9 > ' . 

It should be noted that 'exciept for one day for sickness, each 

. , * ' ' V' ' ' ' ^ ' ^ _ . 

person attended all three days. \ *)r 



Also, two of the participants were from non-school settings. 

' i 

Tliear ^:omments reflect the fact that these materials are directed 

toward a narrowly defined audience. . T^hey were told ahead of tjme 

that the material was very school related, but chose to participate 

anyway. Perhaps this decision reflects the urgent need felt for 

» ^ 
evaluation trailing and the lack 6f such training readily available 

in Alaska. ' ^ ' 

* 

V ' t 

<% 

3S ' • 



Prom my limited oLsert^ation of the workshop, 7 v/ould like to 
add the following comments to those ot the , participants: 

^ ■ . ■ " : 

1; Twelve people was 'a verj^ workable number. 

« f 

Z. The role-playing technique used^ during most of the work- 
shop created good interaction. It seemed to put people* at 
ease. Perhaps people are less kfraid to speak up when 
V they are playing a role. ^ 

3. Three days of attendance is not too much to expect of busy 
people if they feel a need and wish to participate,, and if 
there interest can be held. . v ' 
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. PARTICIPANTS RESPoBlSES i ^ 

. - ^ f ^ ^ ' 4 

U Do you feel you have learned anything in this workshop wnich will 
b/5 useful in your area of responsibility? Explain. 

t f . . 

' This^ workshop was pertinent for me as Assistant Superintendent ojf the 
Notiq* Schools with xesponsi*bility fpr' development, implementart^n and 
evalus^tion of spe^cially f\inde<f programs. . I have ^had the problem of 
having to do too much ojf the proposal development and evaluation with-^ 
out th<e involv'ement of the teaching staff. Traditionally^, the teaching * 
staff has not been involved to any gre^t extent in such efforts. By 
their own admission, inany teachets feel inept in the area of program , 
evaluation* , - i ' ' * . ^ 

1 1 think* the workshop will cause me: . v'*- '\ 

* - To encourage more teacher involvement in eV-altration*^ ^, . \ 

- To spenH time with individuals, and groups explaining evaluatioxi^;^ 
and the process for developing an educatioji evaluation plan. 

- To make evaluation an integral component of^ program planning 
and development. ^ \ * ^ - * 

- To make me aware of.the importance of group dynamics^ i;a the 
evaluation process^ • ,V \ . ^ ^ " 

^ ' - - ^ . 

Definitely! As a present school board memb^ I spend considerable 
time reviewing program evaluations of our own school district as v/ell 
as others. Clearly the seminar has given me expanded perspective 
of the process and product of the evaluation of programs. , 



Although I fel^that I had gone through this type of workshop before, 
there was a definite value in actually working up the kinds of materials 
and to participate in the divisions inherent in the workshop format. 



Yes, I learned something usfe'ful from the workshop* The major steps 
of evaluation for program improvement resembled greatly but was 
superior to my ov/n s^tructur^. The Alternatives /Considerations 
matrix and other methods of presenting data were also useful. 



Yes - As co'ordinator for the minority group activity program, I ' 
feel that the workshop provided me v/ith facts concerning how important' 
it is to have an .evaluation program within a school district, such as, 
who should be involved, the cost, and all the important factors and 
tasks compiled together which can make or break an evaluation program 

Yes. Helped to refocus my thinking re program evaluation. 'Als'o, if 
package becomes available, could use with our staff. 
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.■h Cont'^d'. 



Yes, i am respgr^i^le for programs in elementary ^classrooms, and 
JL need to'have.^a^\Cay inCorm^tion abput,th&se programs that 

» doe s^not cbme^i^ bits^aifa pieces, , . . -J * \ , ' ^ 



^ Yes, I fegl I har^ learned'^^some things from the wo-rkshbp: such as 
XTxeasuring a program and 'how to tovich on both negative and positive 
aspects of a program, also information on sources in the qommtxnity 
to resqa;f to 'for re^^ctioris from tiae people. • \ * ^ ♦ 



^ Somewhat: It's given me a framework/references^for'ove'rall program 
evaluation. ^ * * . • 



/ 



J 



2^ What is your opinion of the materials presented at the \yorkshop? 
a. content? b* format? ' ^ ' 

Theocontent of the materials presented at the workshop were gener^ll)^ 
good. The .materials did not reflect the Alaskan scenB as might 
desirable for a statewide presenilation. Such rhaterials developed 
around an Alaskan scifiool might be a good idea» 

The format was good. I liked th<S idea of using, an actual school as 
a model. The program^iy be a little too loVig to present a^s in-^^; 
service for the average busy school in Alaska. ^ 



/ 

The effective background for simulation activity and reVeal vividly / 
the dynamics of real-life evaluation projects. '^The ^ontent is quite 
good, and I believe the format is effective. Oij^ipossible change , 
^ might be to provide the individual 3?ble descri^ions.on separate 
cards'tp avoid our overviewing other roles to the extent that we 
inhibit that "player's** activity. 



Materials were .excellent in that they represeijted a hypothetical. and 
a practical situation. The content was excellent and the format 
required step-by-step pro'gress. 

1 . , / 

The annotated bibliographies were useful and the additional biblio- * 
graphy was outstanding. Some -negative ixApressions on the format 
were explained in the previous letter. I would have appreciated a. 
one-page agenda or syllabus. • ^ 



Since this was my first time attending such a workshop, I don't feel 
I am completely qualified to elaborate on paragraph 2 of your, question 
naire. I do feel that the material presented was excellent. From 
listening to others in the workshop, the content! could have been 
based on actuality rather than simulation. The format is fine* 



Both good* 



The content was clear and communicated at my level of interest and 
^thinking about evaluation. 
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2. Cont'd , ■ ^, 

'I thqught thd* material given to, us for the amount of time jspent on 
this subject was adequate, because if we got into ahy mor<S, or jnore 
difficult, I would have been confused because- of the lack of tiine • 
spent oil it, , * , - 



• • • 

Much of the content was too educa'tional program-specific for. my 
needs. The fdrmat, especially the simulation exercises/ was strong 
Would help if a,n participants had more, or less the same exposure to. 



subject matter. 



r 



r 



t 
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3» Po you see any use for these materials or similar in your job 
situation? Explain^ • , : • • 

Siinilar Aiaterials will bp used as 1 work directly with the teaching 
staff. The entire program would be hard to- schedule for Nome;, It 
might be acce^tabl^i to a sizeable grt)up.if ip is offered by^the Uni- 
versity of AlaSl^fa^qver a peridd q/, time for credit. If it could be 
offereti as a credit course and me.e^one evening a week for five 
weeks, I suspect ten to" fifteen local peoJ>le will enrolL 



If I do move into, et^ucational administration as I am presently conr 
%sidering^ I'm confident that there will be numerous uses for similar 
^materials. Simulation. ^nd related activities are ra-rely unsuccessful 

m stimulating leai-ning, and J wo.uld seek such materials for any 
'instructional projects for which I was responsible* , > 



I definit.ely see use for these types of materials in our situation in 
Anchorage and talked withthe group leader as to the possibility of 
obtaining copies or permission to use similar items. 



I.have explained previously that much of the content can be adapted 
for evaluation training workshops for a continued Mini-Grants pro- 
gram or for the ANCADA Training Prog-raifi. The format v/ill 
have 'to be altered to suit the target population, however. 

I'm sure this material will be hfe-lpful to me as .a guide for future 
use. It has the necessary instruihents that \vill assist me in deter- 
mining jiow tp go about putting an evaluation team, together within 
my own structure once. I have personnel. 

Yes. Work with professional staff in program ^valuation. 



0, 

Consultants involved in planning programs should be aware of ways ' 
they can be evaluated. 

: 

The only way I, can see me using this material^ in my job situation 
would be comparing the .content with another program that I'm 
working on. • 

^— : i 
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4. In your opinion,, should AEPIC/CNER consider sponsoring 

similar, workshops in the area of program evaluation? briefly 
explain. , ' 

Yes. There is-not enough understanding of evaluation among the 
ranks of the average Alaska school district and additional workshops 
co.uld 'help. . . . . I ' - , , ' ' 



Yes, I believe AEPIC/CNER could provide a real service - especially 
to small districts in Alaska - by sponsoring sixrillar workshops in. 
the area of program evaluation, Corisumers seem. to be asking for 
•more evaluation, and I feel certain that'we are entering an era of 
greater emphasis on this field. 



It would appear to me that there are defi;iite advantages ifi having • 
an agency or institution set up these types pf workshops. Then there 
is no question as to the reasons for holding them. ' That is, people 
sometixpes question the motive of a single district or state agency. 
I do not feel that the University would be so questionedo. 

Ilm really unable to offer any advise on this question,, I gained from 
the workshop, but I kno\y that this program is not one ,that you're 
required to service. f - , * 



IJbeiieve workshops such as this one would be beijeficial in this areg.. 
It should be made available to administrators, teachers and other 
concern groups. Our evaluation program needs strengthening, .and 
this is one way to do that. 



/ 



4- 



Yes. Larger districts could undertake this on their own, however. ^ / 



• Ye^. Either- at the next step for this group or as an introductqry 
session like this one for a new group. Would personally like .the 
"next step''. . 



Yes, I think a program evaluation wo;rkshop shoilld take place so we 
can cover the evaluation of many types of. programs. 



' Yes, I'd like to see a workshop designed around a more general ^ 

approach to progr^am evaluation (less secondary^ urban education 
. context). 



0" ♦ 
f 
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PleaS^ comm|nt £r£iel/'as to any additional reactions,^ugges tions 



oetc 

o 



x'xnay havd. 



/J 



^ Dr. Earl Mortensen^/as an excellent leader. I question if the materials 
pould be presented agcurately by someone not as krjiowledgable regard- 
ing th6^ pi^ogram^as was Dr. Mprtensen. He indik:atjed that with some 
in-servi<fe. trailing,, almost anyone could cphd^ct the pjrogram'using 
the prepared materials, but I think screening of jhe , candidate for 
leader would. b^ essential. . 



/*If the content of the workshop were condensed to a two -day schedule, 
including an evQ^ning, some districts niight release employees ior 
ft more readily. . . ^-^ ^ 



I think lhe^at>ove represents my feelings, and would. add no more. ^ 

0 ? — • 

Perhaps an "instant replay" of some of the simulation activities 
wolild be useful, in stimulating rcioi'e serious^participatioh. ."^ 

As DeLJohnson arid Lou Gonzales would have it, lets sximmarize the 
progf^am and assigri tasks. " 

I enjoyed the workshop and the personnel involved. Inner changing 
of ideas are very useful* you know. 



^/3ood workshop. Enjoyable, meaningful aiid helpful* 



Excellent -.1 would have profited by concluding statements from the 
moderator, even, though it was .contained in the written rhaterial I 
respond best to "live" input.''. . • 



I feel that ju^t about anjr evaluation workshop is worthwhile l^eiijg' 
^that it is organized properly. • ■ - . " 



wfiat 



More an how tp define what is measurame (goals and objectivesjjj how 
to^ measure then^ (models) - I really wanted something more general, 
y ^ which would readily apply to a vai:iety of programs (this niight not 
fall ihto the realm of AEPIC/CNER but I think it should! ). . 



ERIC 
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LIST OF PARliieiPANTS 



'ARliiei 



^Evaluation Workshop; Evaluation for Program Improvement 
• " October 14-16, Holiday Inn, Ancborage 



Greater Anchorage fiorough School^ District 

670 Fireweed Lane, Anchorage, AK 99503 
277-66^^ 

Dr. William Marsh 
Ms. Susan Greene 
Dr. Robert Vanslyke 
. Dr. Anna Beth Brown 



"A 



Fairbanks North Star Borough School District 
\ P. O. Box 1250* Fairbanks, AK 99701 
\456-7934 \ *^ 



•Mr. Lee C^urie 
Mr. '^Willia/n Oates* 



Alaska Federation of Natives, Inc. 

1675 ''C' Stfeet, Anchorage, AK 99501 
274-3611 



Technical Assistance 



Mr. Eric Ekvall 



JOM -?-E valuator 

^ . Mr. Karl Qreenewald, Jr. 

^ ■ yis\ Addie Brooks ^ 

' Ms. Bertha Lowe 

ANCADA 

528 West 5th Street, Anchorage, AK 99501, . 
277-2578 " ^ 



Mr. Michael Moore 



Nome City Schools 

P. O. Box 131, Nome, 'AK 99762 



Mr. /D&rold Hargraves 
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APPENDIX C 
Questions for Expert "Reviewers of 



Evaluation for Program Improvement 



In your review and report, you are encouraged to resf)orid to the « 
kinds of questlorfe you miglit ordinarily ask In review of training 
Inaterlals being considered for use by you in your schpol setting, 
or in a training session yoa might be conducting for*-others. We 
are seeking your judgments about the unit, made in a way that you 
think* is most re^alistic and meaningful to you as a user based on 
your own pa5t experience with similar kinds of materials. However, 
listed below are the questlQns to'which we would lik§ you to 
' respond; 

• * '1. Do- the goals and objectives as stated in the unit,- or as 
inferred by you from the content, afddress an impbrtant need for 
school staff, which if met would result in improving the effec- , 
tiveness with which school staff perform in their jobs? 

2. Recognizing that there will be some relative minor 
revisions made before the unit is released, does the unit in its 

^ present form and content seem to be well directed^ to the stated 
- .goals and objectives? 

3. Is there additional information or materials that you 
believe would simplify or improve the work that the unit Coordina- 
tor would have to do? Are there situatii)ns or kinds of Coordinators 
for which you might predict the unit would, not work, and if so^ can 
you describes the limits on its use? (We know that the unit can 

be used effectively by non-Laboratory Coordinators, because it has 
been done. What we don't know, and have neither time nor money 
to determine, are th? limits on effective u^e of the unit.) 

4. If the unit were ta be used, but the staff could not devote 
the full fifteen hours judged to be necessary, what would be your 
recommendations for sections to be eliminated, stated approximately 
in priority *rder? (What coiild^e eliminated first, what second, , 
and so on. ) 

/ \ 5. Do you believe, that this unit would bemused in school 

' staff training if it were available at a cost of $8.00 per partici- 
pant, plus a charge of $35.00 for Coordinator's materials? 

6. If a school district sought your recommendation as a paM 
consultant for a staff training program, and you were satisfied that 
, ' staff was clear on what they wanted to accomplish in a new program, 
would you recommend use of this untt? What other training procedures 
and materials might you consider a& alternatives to this one? "How 
would you rank this unit relative to these other materials? 



RESPONSES TO SPECIFIC QUESTIONS REGARDING 
. TRAINING MATERIALS FOR EVALUATORS PRODUCED 
BY FAR WEST LABORATORY FOR EDUCATIONAL RESEARCH: 
AND DEVELOPMENT 

DAVID A. PAYNP 

• * 

1 . Adequacy and Responsiveness of Unit Goals and O&jectives ;- . 

It is difficult to identify wfiich individualCs) in a school setting would 
have the major respQjisibility for undertaking program evaluations. In larger 
systems there maybe an "official" evaluator. or office of research. For personnel 
in these kinds of positions, or rather the second line personnel, the unit maybei 
appropriate. This revievrer definitely feels that the evaltiation unit would be 
appropriate for principals and instructional supervisors, and to a lesser extent 
for teachers. 

The goals listed on. pp. ix-x, and objectives identified on pages 1.3, 2.3, 
3. 3, U»3, and 5.3 are definitely important ones, and if met and whjen applied ^ 

could result in improved instructional programs. One vronders if objectives 

V ' ' 

dealing with the mechanics of undertaking cost-effectiveness analyses have been 

slighted. • ' * . 

The five steps identified in the evaluation process are logical and the ones 

usually js tressed in such training piiograms. 

2, Relitidn of Unit Activities to Goals and Objectives. 

• In G^eneral activities are highly responsive to the session objectives. , One 
possible exception might be the degree to which Objective 2 on p. 2.3 and Objecti\ 
2 on p. "3.3 are, treated adequately in their respective sessions ♦ 

Obviously great care has been takerf^in the development of the materials. 
The Jirections for the coordinator are the most complete this reviewer has seen. 
The raateriai 6i> r£f^le playing and discussion-leading is most helpful. In addition 
tthe explicit and implicit emphasis on decision-making is to be commendedl. 
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3. Tasks of the Coordinator , - / 

. ' / 

Obviousiy some background in research, measurement, and evaluation, both ' 

/ 

academic and experiential, uould be most helpful to a potential coordinator. 
In addition some human relation training would stand a coordinator in good ^ 

t 

Stead. 

(In addition see attached Supplementary Resources List) 
Training Time Devot§'d to Unit and Sections / 
In general tasks seem relative^ly well balanced for time, yds reviewer . 
would».be very hard pressed to eliminate any of the sessions^ ^The notion, as 
-suggested on p. xv of using a 12h or 13^ hour schedules ^p4s not seem like 
a viable alternative. The saving of tiiTie is not that great, and yet consi^derable 

i 

loss in training over important topics would probably result. 

Despite the fact that they^have been field tested thiS' reviewer finds it 
difficult to accept the fact that Sessions 2 and 3 each really require 
three h^urs. 
5. ^ Gbst of Materials . 

** .There is no doubt that the projected pricing of the unit will be one of the 
' strojig points in^^t^s appeal. Most other similar units, particularly those 
available. frpm commercial sources, are priced three or four times the present 
price/, tosts are well within the budgets of virtually all p^iblic schools and 
tho^.e of graduate students. 

6... Use of Unit ' "^^v^ 

This reviev/er .would have no reservations about using the unit either if* a 
college or university teaching situation or in conducting workshops with public 
school, personnel or prv^f essional organizations (assuming that they were relatively 
inexperienced in evaluation). 



f > 0 
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A list of additional reaoarce materialsi that might be used by t|iB coordinator 
to gain background information is enclosed. 

i ' ^ 

At'-this point in time 'the chief 'Competitor'' with the Far tfe^t materials 
would probai;^ly be the simulation exercises developed b^ Blain Worthen of the 
Northwest Regional Educational L^oratory and others, and published by Charles 
Jones Publishing Company ,of Worthington, Ohio.^ These materials run into several 
hundreds of dollars and are not 'nearly so detailed in presentation, particularly 
with regard what participants and coordinator are to do, as are the Far ^est 
materials. The Far West materials are far superior with regard to objectives • 
and specification of activities-, ^responsibilities , and roles of both participants 

N 

and coordinator. 

7. Miscellaneous Observations 

Ca) It is not clear in the Introduction that the Annotated Readings are for 
the Coordinator and not the participants. 

(b) The Prologue will probably not rece:ive an Oscar'. 

(c) Is giving the participants a choice of roles to play a reasonable approach? 
It is obvious that some will be disappointed.^ Why not just allow the 
coordinator to assign roles to individuals on the basis of his 'knowledge of 
their backgrounds? 

(d) Excellent summary of data collection methods pp. 3.33-3.37. 

(e) The suggested reading in Stufflebeam et.al. of Chapter U is perhaps ill- 
advised due to complexity and high degree of ab,stractness. ' It will i^ot 
win a Pulitzer Prize for Literature. 

(f) At times it is. difficult to follotf transitions from coordinator activities 
or materials 'to participants activities and materials, e.g from p.l.2U 

to .1.25. 
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(g) The scoring guidelines for the wrap-up activity of pp. 5,28-5vU2 are ' 

most helpful* - ' ^ 

(rj One wonders if it would be possible to provide participants real practice 
in summarizing, analyzing, reporting, and writing recommehdations from 
hypothetical data. This kind of activity could perhaps be included 
between the Fourth and Fifth session, 
(i) Dr, Hartman vrill i^^eally have to be on his or her toes during the Fifth 
s^ession due to the magnitude and coinplexity of the exercise. Coordinator 
needs to work very closely with this person prior to session, 
(j) Providing participants v/ith a hypothetical budget, or having them generate 
one would add greatly to the realism of the unit. Particularly when, 
the budget reduction is imposed, A percent reduction would have to be 
specified. 

(k) V/ould it be possible to gd^ferate a time matrix like that on page 1,3 

for all the exercises and sessions so that the coordinator could better ^ 
plan for time? 

(1) It is difficult to' really get a handle on what "considerations*' are 
in Session 3. Are we really not talking about dependent variable 
class if icaticfns? I would not use the term dependent variable, but 
* would consider the concept. The intersection of source and variable 
classification categories could still be used to specify instrumentation. 

(m) Could not some of the participants in-session reading be accomplished 

«- 

>' 

prior to meetings thegeby allowing for shorter sessions? The material 
in the participants handbook on pages 96-98 might pjrof itably* be read, 
for example, prior to Session 3. 
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